AI Assistant Outage
Incident Report for tawk.to
Resolved
We're pleased to report that the restoration of the Vector Database is now complete. Full access to all knowledge sources should now be restored.
Posted Nov 29, 2023 - 12:29 UTC
Update
There were unexpected issues during the 'Website Knowledge Source' restoration process, causing a delay. However, the dedicated team has efficiently addressed and resolved these challenges. The restoration process is now back on track, and it is expected to be completed within approximately one hour.
Posted Nov 29, 2023 - 07:12 UTC
Update
We wish to update you on our ongoing efforts to restore the 'Website Knowledge Source' in our Vector Database. At present, we have successfully restored close to 30% of the overall 'website knowledge', and we are fully committed to achieving complete restoration as promptly as possible.
Posted Nov 28, 2023 - 16:13 UTC
Update
We are pleased to report significant progress in our restoration efforts. The AI response functionality has been fully restored, and the 'KB (Knowledge Base) knowledge source' is now operational again. Our team is currently focused on the restoration of the 'website knowledge source'.
Posted Nov 28, 2023 - 12:56 UTC
Update
The restoration process for both the 'KB (Knowledge Base) knowledge source' and 'website knowledge' is currently underway. We are making good progress and are optimistic about a swift completion. Additionally, we are in the process of re-enabling the AI Assistant to provide its full range of services.
Posted Nov 28, 2023 - 11:50 UTC
Update
We are pleased to announce that the 'document knowledge source' has been successfully restored in our Vector Database. Our current focus is on restoring the 'KB (Knowledge Base) knowledge source' and the 'website knowledge'. Our team is working tirelessly to complete these tasks and fully reinstate all services.
Posted Nov 28, 2023 - 10:53 UTC
Update
We have successfully restored the 'plain text knowledge source' in our Vector Database. Our team is continuing to work diligently on restoring the 'document knowledge source'. We are committed to resolving this remaining issue as quickly as possible and will provide a further update soon.
Posted Nov 28, 2023 - 06:30 UTC
Update
We are actively working on restoring the 'document knowledge source' in our Vector Database. Our team is dedicated to resolving this as swiftly as possible. We will provide a further update soon.
Posted Nov 28, 2023 - 03:00 UTC
Update
We have successfully restored the 'shortcut knowledge source' index in our Vector Database. We are now working on restoring the remaining knowledge sources. Progress is being made, and we anticipate a full resolution soon.
Posted Nov 28, 2023 - 00:00 UTC
Update
The initial fix attempted for the Vector Database issue was unsuccessful. We are now implementing a secondary plan, which involves re-indexing to a new cluster. This approach is aimed at resolving the problem more effectively.
Posted Nov 27, 2023 - 21:00 UTC
Update
We have identified the root cause of the issue with our Vector Database. Our team is now actively working on applying the necessary changes to fix the problem.
Posted Nov 27, 2023 - 19:00 UTC
Update
Currently addressing an issue with our Vector Database in collaboration with the vendor. We are working diligently to resolve this as quickly as possible and apologize for any inconvenience caused.
Posted Nov 27, 2023 - 16:00 UTC
Update
The issue has been identified as originating from our Vector DB vendors. A support ticket has been raised with the vendor, and we are currently awaiting their response.
Posted Nov 27, 2023 - 15:30 UTC
Update
Upon thorough investigation, the root cause has been traced to our Vector DB, which is currently not responding to queries as expected. This is directly impacting the functionality of the AI Assistant addon, leading to the observed increase in API errors.
Posted Nov 27, 2023 - 15:15 UTC
Identified
We have identified a critical issue affecting the AI Assistant addon, marked by an elevated rate of API errors. This incident was initially detected around 1420 UTC today.
Posted Nov 27, 2023 - 14:45 UTC
This incident affected: AI Assist.