With the downtime experienced overnight, we want to change the protocol for how critical events are handled. 


What is a critical event? 


These are incidents whereby clients' ability to either fund their accounts or trade them is restricted in some form. The obvious examples are: 

  1. The hub is not responsive and cannot be logged into 

  1. The hub is responsive, but clients cannot access the deposit screen 

  1. The trading server is down, and clients can't connect 

  1. The trading server is online, but no pricing is coming through to the platform 

  1. The trading server is online, and pricing is available, but clients can't trade 

In the event that a critical error has taken place and the Melbourne office is offline, we would like the PH team to contact the following team members and ranked in terms of who to call first.  


  • Emergency Phone (Currently held by Chris): +61432346878 

  • Chris Hewitt: +61403067652 

  • David Swinden:+61413290522 

  • Phil Horner: +61419881215 and +61405266093 


Since our phones might be on "Do not disturb" mode, each number will need to be rung twice immediately after one another.  


Note:

Please send a message to teams with the details first before calling the emergency phone - Chris H.


What if I still can't get a hold of someone? 

If the critical event has been going on now for longer than 10 minutes and you haven't managed to get a hold of one of the members of the team above, we will need team PH to reach out to our main providers directly to inform them of the issue to see if there is a way that they can assist. 


Hub/Deposit Issues: 


If we are facing a critical issue with our hub, team PH will need to contact our development team directly at: teamfusionmarkets@coherentsolutions.com. 
 

This will go to all the developers simultaneously and should receive a quick response when the Melbourne office is offline. You would write the email to them in the same manner when you are reporting a bug to the Melbourne office. In this instance, though, the subject would be Urgent: <insert issue>. Rather than High Prio: <insert issue>. 
 

Trading Server/Pricing Issues: 

Should there be a critical issue with the trading server, team PH will need to contact our service provider: support@onezero.com. 

The team at OneZero manage our trading infrastructure, including our trade servers and connections to various liquidity providers. If there is an issue with our trading environment, they are best placed to resolve it. 

Where they have found an issue but believe that they cannot fix it, please continue to try and contact Chris and me until someone answers the phone. 

An example like this might be that one of our liquidity providers has stopped pricing. OneZero would be able to inform us that this is the issue, but they can't fix this on their end as it's not their issue. 
 

What next? 

Hopefully, this isn't something we need to use often. Our systems/applications have been relatively stable for a while now. Problems will undoubtedly occur in the future, and we also have tools to assist and alert us when things aren't working as they should. 
 
As it turns out, the developers were already aware of the issue when I messaged them last night and were working on resolving it. With this process, though, they would have been alerted to the problem a lot sooner, decreasing the total amount of downtime. 

Giving you the ability to contact our external providers should provide you with confidence that issues are being worked on and resolved and things will be back to normal ASAP.