Status Updates

Lecture Recording @ York - Degraded Service

Incident Report

 

Resolved

(Feb. 5 2017, 12:15pm)

 

 Since the upgrade and fix on Friday there have been no further incidents or issues with recordings being published as expected. All but a couple of recording have now been published, and we expected to have to address them manually and will do so first thing Monday morning. We're now considering this issue resolved and this will be the final update. Many thanks for your patience as we've worked with the vendor and UIT teams to rectify this issue.

Monitoring

(Feb. 4 2017, 11:00am)

 

 Publishing continues without incident. There are currently 60 recordings still in queue to be encoded and published.

Monitoring

(Feb. 3 2017, 6:00pm)

 

 The server continue to publish recordings as expected. We continue to monitor. We are making progress on backlogged recordings. Along with the new recordings added today we currently have 120 recording in the publishing queue.

Update

(Feb. 3 2017, 10:30am)

 

 We have upgraded the Lecture Recording server to implement the transcoding fix. The upgrade appears to have rectified the transcoding issue and recordings are once again being processed. We are monitoring. At this point there are more at 140 recordings to process.

Update

(Feb. 2 2017, 4:00pm)

 

 We are currently in final testing of the fix to our continuing transcoding issue which has been preventing the publishing of recordings submitted to the Lecture Recording server. We have arranged to upgrade our server with this fixed about about 9am tomorrow morning (Friday). This upgrade will not affect any of your recordings though there may be a delay in the computer you're using submitting them to the server. During the upgrade you will also not be able to log into the server to check on the status of your recordings. In short, please continue recording as usual during this upgrade.

Update

(Feb. 2 2017, 1:00pm)

 

 Working with our Lecture Recording vendor Techsmith, we have identified an issue with one of our transcoding components that had been causing publishing stalls and job failures. The have developed a fix for this issue and we're working with them to qualify the fix. We are also working with them to determine alternatives to allow publishing of recordings through other means while we address the server issue. We hope to address both these issues in another update later this afternoon.

Update

(Jan. 31 2017, 11:00am)

 

 The publishing of lecture recordings continues to be very slow. Our product vendor, Techsmith, has been analyzing our very high volume data logs and is in the process of providing some remedial actions. We will update the community as soon as possible when service is restored to normal.

Update

(Jan. 29 2017, 11:00pm)

 

 Most recordings have now been published. However, twenty-five recordings resulted in errors that prevented their publication. We have tried resubmitting these recordings a number of times without success. We'll be engaging with Techsmith, our product vendor, first thing in the morning to address both this issue and the slower than desired publication speeds we've seen since Thursday.

Monitoring

(Jan. 28 2017, 10:30pm)

 

 We continue to monitor publishing performance which is a bit slower than we would like. Two thirds of recordings submitted since end of day Wednesday have now been published.

Monitoring

(Jan. 28 2017, 9:30am)

 

 The Lecture Recording service continues to catch up with the publishing of recordings. Half of recordings submitted on Thursday have now been published. We continue to monitor performance and progress.

Monitoring

(Jan. 27 2017, 2:10pm)

 

 The Lecture Recording service is running normally and is now accepting recordings from earlier in the day. As computers that created the recordings connect to the internet they will begin to submit the recordings to the server automatically. You do not need to take any additional action. You may check the status of your individual recordings by using your Passport York account to log into the server. There is quite a backlog, so it will be many hours before all recording are published as expected.

LTS will be monitoring the status of recordings this evening and over the weekend and will provide updates, though less frequently. We appreciate you patience as we recover from this issue.

Update

(Jan. 27 2017, 2:10pm)

 

 Publishing services are running as expected. We are processing yesterday's recordings at the moment and expect to open the server to accept those recordings made today within the next 60-90 minutes. If you've created a recording no action will be needed on your part.

Update

(Jan. 27 2017, 12:20pm)

 

 Publishing services are now running as expected. We are not yet accepting new recording submissions as we continue to monitor publishing performance of queued recordings.

Update

(Jan. 27 2017, 11:15am)

 

 UIT service teams have completed the server upgrade. LTS is now testing recording publishing performance.

Investigating

(Jan. 27 2017, 10am)

 

 We are investigating a severe degradation in publishing performance of the Techsmith Relay lecture recording solution.  At this time most recordings sent to be published to the server yesterday (Thursday, January 26, 2017) are still in the queue and awaiting processing.

This issue does not affect the creation of recordings inside or outside of the classroom.  Please continue to record and submit your lectures as usual and they will be processed once we’re back in service.  We have been in touch with the vendor and are preparing an emergency minor version upgrade to the service which the vendor believes will rectify this situation.  More information on the timing of this upgrade and effects on service will be provided on this page.