tag:status.orionlabs.io,2005:/historyOrion Labs, Inc. Status - Incident History2024-03-28T05:25:10-07:00Orion Labs, Inc.tag:status.orionlabs.io,2005:Incident/190868662023-11-10T12:07:37-08:002023-11-10T12:07:37-08:00Orion Platform Experiencing Degraded Performance<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>12:07</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:34</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented currently monitoring the results.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>11:06</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>10:58</var> PST</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.orionlabs.io,2005:Incident/188686882023-10-20T11:16:01-07:002023-10-20T11:16:01-07:00Orion Platform Experiencing Degraded Performance<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>11:16</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>09:53</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented we are currently monitoring the status.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>09:15</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and the fix is being implemented.</p>tag:status.orionlabs.io,2005:Incident/187155782023-10-07T01:49:05-07:002023-10-07T01:49:05-07:00Orion Voice Platform Experiencing Degraded Performance<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>01:49</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>01:39</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented. We will continue monitoring.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>01:21</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>01:20</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.orionlabs.io,2005:Incident/186173332023-09-26T20:39:27-07:002023-09-26T20:39:27-07:00Orion Services Has Been Restored<p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>20:39</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>19:46</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>19:43</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are now monitoring the results.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>19:36</var> PDT</small><br><strong>Identified</strong> - The issue has been identified a fix to be implemented.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>19:10</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.orionlabs.io,2005:Incident/182483562023-08-24T13:25:08-07:002023-08-24T13:25:09-07:00The Orion Platform is Experiencing Degraded Performance<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>13:25</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>12:18</var> PDT</small><br><strong>Monitoring</strong> - One of backend services was causing degraded service. The service was restarted and performance returned to normal. We will continue to monitor the results.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>10:20</var> PDT</small><br><strong>Identified</strong> - The issue has been identified, fix is being implemented.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>09:51</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue</p>tag:status.orionlabs.io,2005:Incident/177935482023-07-15T00:00:21-07:002023-07-15T00:00:21-07:00Orion Voice Platform Maintenance<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>13:23</var> PDT</small><br><strong>Scheduled</strong> - During this maintenance window, we will be performing upgrades to our database infrastructure to enhance system performance and reliability.<br /><br />Please note the following important information regarding the maintenance:<br /><br />Impact on User and Group Settings: We kindly request all users to refrain from making any changes to their user and group settings during the maintenance window. Any modifications made during this time will not be saved. To avoid any potential issues, we recommend making any necessary changes before the maintenance window or after it has been completed.<br /><br />PTT Message Functionality: We want to assure you that the maintenance activities will not affect the majority of Push-to-Talk (PTT) messages. Most PTT messages will continue to function as expected, and most messages sent during the maintenance window will be saved and archived properly. A small minority of messages within a 2-minute window in this period may be lost as we cut over to new systems. We recommend re-sending any messages that are not acknowledged by your intended recipient(s).</p>tag:status.orionlabs.io,2005:Incident/174648852023-06-03T15:01:48-07:002023-06-03T15:01:48-07:00The Orion Platform is Experiencing Degraded Performance<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>15:01</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>14:30</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented, the team is currently monitoring the results.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>14:20</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fixed is being implemented.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>12:35</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>12:12</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.orionlabs.io,2005:Incident/163937172023-03-23T14:12:10-07:002023-03-23T14:12:10-07:00Message Archive is Now Available<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:12</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. Message Archive is fully functional and working as expected.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>09:24</var> PDT</small><br><strong>Monitoring</strong> - The issue has been identified and a fix has been implemented. We will continue monitoring the fix. Users can now use Message Archive in the Orion Command Center.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>09:52</var> PDT</small><br><strong>Update</strong> - We are continuing the investigation and will have a status update on March 23rd.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>14:31</var> PST</small><br><strong>Investigating</strong> - We are currently investigating the issue. If Message Archive information is needed please reach out to support@orionlabs.io with date and time needed and we can manually provide for the time being.</p>tag:status.orionlabs.io,2005:Incident/163580732023-03-06T12:30:20-08:002023-03-06T12:47:51-08:00Emergency maintenance due to potential vulnerability<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>12:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>11:16</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:53</var> PST</small><br><strong>Scheduled</strong> - We have identified a potential vulnerability and will be applying a change to address and ensure secure service. Service may be degraded for 10mins during the window.</p>tag:status.orionlabs.io,2005:Incident/126864262022-11-02T21:00:45-07:002022-11-02T21:00:45-07:00Orion Voice Platform Maintenance<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>21:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>12:56</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.orionlabs.io,2005:Incident/108384772022-08-17T19:52:30-07:002022-08-17T19:52:30-07:00Orion Voice Platform Maintenance<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>19:52</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>08:40</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance at this this</p>tag:status.orionlabs.io,2005:Incident/108354322022-08-16T00:17:55-07:002022-08-16T00:17:55-07:00Orion Voice Platform Experiencing Degraded Performance<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>00:17</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>00:07</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>00:01</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.orionlabs.io,2005:Incident/106568572022-07-28T22:00:30-07:002022-07-28T22:00:30-07:00Orion Voice Platform Maintenance<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>17:30</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.orionlabs.io,2005:Incident/106556412022-07-20T17:27:18-07:002022-07-20T17:27:18-07:00Orion Voice Platform Maintenance<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>17:27</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>16:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>13:08</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. We expect the platform to have no downtime.</p>tag:status.orionlabs.io,2005:Incident/105694932022-07-20T05:01:07-07:002022-07-20T05:01:07-07:00Orion Voice Platform Maintenance<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>05:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>02:02</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>12:45</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. We expect the platform to have no downtime.</p>tag:status.orionlabs.io,2005:Incident/106299952022-07-18T18:32:56-07:002022-07-18T18:33:07-07:00Orion Command Center: Groups and Dispatch Functionality - Partial Outage<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>18:32</var> PDT</small><br><strong>Resolved</strong> - The outage was caused by an unexpected spike in traffic which caused the degraded performance of our systems. The issue has been resolved and a fix has been implemented. All systems are now working as expected.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>18:05</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>16:46</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate the issue. Please note - Dispatch users that are logged in, please do not log out and log back in or reload the page as it will prevent you from accessing the Dispatch console.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>15:02</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.orionlabs.io,2005:Incident/96295162022-03-25T11:36:39-07:002022-03-25T11:36:39-07:00Orion Admin Function: Message Archive - Degraded Performance<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>11:36</var> PDT</small><br><strong>Resolved</strong> - The issue has been resolved, and Message Archive is functioning as expected.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>11:31</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and we are implementing a fix</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>09:28</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue</p>tag:status.orionlabs.io,2005:Incident/92675562022-02-08T11:01:43-08:002022-02-08T11:01:43-08:00Partial Outage on Orion Platform<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>11:01</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>10:01</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the fix</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>09:51</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>09:45</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue that is providing intermittent degraded service on Orion Platform</p>tag:status.orionlabs.io,2005:Incident/92245852022-02-02T16:57:44-08:002022-02-02T16:57:44-08:00Orion Voice Platform Outage<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:57</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>15:27</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented a fix and are monitoring for quality assurance.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>15:01</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.orionlabs.io,2005:Incident/91706222022-01-26T12:59:14-08:002022-01-26T12:59:14-08:00iOS devices not receiving incoming audio messages while app is backgrounded<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:59</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>11:49</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>11:44</var> PST</small><br><strong>Identified</strong> - We have identified the issue and working on a solution.</p>tag:status.orionlabs.io,2005:Incident/90838752022-01-26T01:00:29-08:002022-01-26T01:00:29-08:00Orion Voice Platform Maintenance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>01:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>23:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>11:01</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance at this time.</p>tag:status.orionlabs.io,2005:Incident/88675992021-12-16T15:19:37-08:002021-12-16T15:19:37-08:00Partial outage Orion voice workflows<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>15:19</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>14:58</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring results.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>13:58</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>10:46</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.orionlabs.io,2005:Incident/87863922021-12-08T08:49:43-08:002021-12-08T08:49:43-08:00AWS Outage Impacting Orion Services<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>08:49</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved. Amazon Web Service has recovered.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>13:50</var> PST</small><br><strong>Identified</strong> - There is currently an Amazon Web Service that is impacting Orion Services. Users may experience delayed messages or missed messages on iOS devices.<br /><br />The issue has been identified by AWS and they are working toward a fix.</p>tag:status.orionlabs.io,2005:Incident/82661392021-10-19T13:51:38-07:002021-10-19T13:51:38-07:00Severe Service Degradation on all Clients<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>13:51</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>11:02</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:56</var> PDT</small><br><strong>Identified</strong> - The issues has been identified and we are implementing a fix.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>10:23</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating</p>tag:status.orionlabs.io,2005:Incident/80486102021-09-21T16:06:32-07:002021-09-21T16:06:32-07:00Orion PTT Outage<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:06</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:05</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>12:59</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the result.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>11:35</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>