Vista Service Stable; Next Steps

Blackboard Vista service has been heavily used and stable since being restarted twelve hours ago.

Over the next two weeks IRT will conduct a root cause analysis for the Vista database corruption problem.  Broadly speaking, we already know that the replacement of a failing piece of equipment caused the problem, but we don't yet know why doing so would cause such an issue as the entire infrastructure is engineered with full redundancy in mind.  We have numerous email threads to review, service logs from five different hardware/software products to correlate, and 12 staff (the recovery team) to interview.  This process is essential to finding out precisely what caused the computing environment to lose integrity.

After the root cause analysis is complete IRT will review system configurations, product mix, and operational procedures to determine what additions or changes need to be made to both avert future problems and to reduce recovery time.  This review process will take two to four weeks to complete.

We'll have more information for the community in mid- to late-December.

Blackboard Vista Service Restored

Normal Blackboard Vista service has been resumed.  As of 7am today, the database is as it was at 7am on Sunday, before the problems began.  Information entered while the system was in "read-only" mode on Monday is not available because it is in the damaged database that is offline.

Repair Complete; Restarting Vista

The Bb Vista database has been repaired to the state that it was in on Sunday morning at 7:00 AM EST.  The damaged database was shutdown at 4:45 AM and the repaired database has been attached to the server.  The server is now being restarted.  IRT staff will conduct some testing to ensure that Vista functions are working again and the system will then be opened to users.  Normal service should be restored by 7:00 AM if all goes well.

The damaged database is offline.  Information entered into Vista while it was running in "read-only" mode on Monday and early this morning is not available. 

Stage 2 done; More Repairs and Planned Outage Tonight

Stage 2 recovery sped up significantly as it processed files and it finished at 9:30pm, over six hours ahead of schedule.  We started the final stage of recovery, but encountered a snag and are working with Oracle to get around it. Until we get passed this, we can't give any updated time estimates.

We expect to shut down the damaged Bb Vista server at some time between 10:30 and 11pm tonight sometime over night.  Once we do shut down, the system will be unavailable for about two hours while the recovered database is attached to the Vista server.  If all goes well, Blackboard Vista will be back to normal around 1am on Tuesday morning.

Edited at 10:40pm.

Stage 1 Complete; Stage 2 in Progress

Stage 1 completed successfully and stage 2 is running.  It is currently very difficult to estimate how long stage 2 will run because each of the 217 files that must be processed is taking a different amount of time.  The current estimate is that this stage will be complete at 5:00 AM on Tuesday, but this is still a very rough guess.

After stage 2, there is approximately 4 hours of work that must be done.  Two hours into that window, the Bb Vista that is currently running with a damaged database will be shut down -- this looks like it will happen around 7:00 AM.  About two hours later, the fully-recovered system will be available to users -- this looks like it will happen around 9:00 AM Tuesday.  All of these times are extremely rough estimates and will be updated later tonight.

Engineering Review Complete; Stage 1 Finishing

The technical support engineering review of the almost-complete Stage 1 processes has been completed and found to be OK.  The final parts of Stage 1 are running now and expected to complete in about an hour.  However, there are still two stages to go.

Oracle Engineers Reviewing Work

Support engineers from Oracle are now reviewing the state of the database before we procede with further steps.  We'll let you know when there's new information.

Stage 1 of 3 Almost complete

Stage 1 of the 3-stage database recovery process is expected to complete by 4pm.  We will then have a working copy of the Bb Vista database from Friday morning and will begin stage 2 to apply "redo" logs to update the database to the state that it was in at 6am on Sunday, an hour or so before the crash.  Thee third stage will bring the database more and more current to just before the crash and the data corruption.

Once we have completed all three stages – likely early this evening – we will bring Vista down, attach the recovered database to it, and restore normal service.  This planned outage will take about 30 minutes.  We’ll let you know an estimated time for the outage as soon as we can.

Notice to Users regarding Bb Vista

As you likely know by now, the Blackboard Vista system sustained an extended outage on Sunday. While service was restored on Sunday evening, additional errors occurred over night and it is not in a sustainable state.  Bb Vista use should be avoided today, if possible.

Cause:

On Sunday morning, the Vista server encountered a problem that caused the database to become corrupted.  IRT worked with Oracle technical support engineers to restore service.

Status and User Instructions:

The system was brought up on Sunday evening.  It is running, but additional database problems were encountered over night and Oracle now advises against its continued use.  Due to its critical importance, we are leaving Vista up in this impaired state while recovery operations are underway.  We advise all users to treat the system as if it were “read-only”.  As such, you may read content and review existing materials, but do not add content, create or submit assignments, conduct tests, do grading or participate in discussions.  All activity from today will likely be lost when the recovery is complete.

Plans:

A database recovery operation is currently running on a backup server. When that finishes, we will have a Vista database that is back to the state that it was before the crash.  Once we have that – likely early this evening – we will bring Vista down, attach the recovered database to it, and restore normal service.  This outage will take about 30 minutes.  We’ll let you know an estimated time for the outage as soon as we can.