All,
some good news. Zoom believe that they have identified the root cause of
the problem.
It has to do with that the recording connector needs to migrate files
during processing.
(If the disk is busy at the time the file movement might fail and this is
resulting in a “dirty” file, and this also affects subsequent files and is
the reasoning for the generic slowness.)
Zoom is working on a new release of the recording connector, that will be
deployed as soon as available.
Brgds,
Jørgen Qvist
Det kan ta upp till 48 timmar innan en recording dyker upp men *vår
rekommendation (och NORDUnets) är att stänga av funktionen tills problemet
är löst. *Vi har kommunicerat detta i omgångar, men det trillar fortfarande
in ärenden om förlorade inspelningar.
Vill ni vara helt säkra på att det fungerar så stäng av på kontonivå. Vi
vet att både NORDUnet och Zoom jobbar på problemet.
Alltså, stäng av funktionen för cloudrecording i Zoom tills problemet är
löst.
Mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej!
Detta kom nu på morgonen.
"All,
There is nothing new in regards to the status, the root cause has still not
been identified. There is a lot of effort going in to finding the cause of
the intermediate problem, both by Zoom and NORDUnet and we have found other
things that have been fixed along the way.
There was an issue with Zoom that affected the Zoom users in Finland and
parts of Sweden Thursday afternoon. This was cased by some local VM disks
running full due to the increased logging enabled to debug the recording
issue. The problem was quickly resolved.
Brgds,
Jørgen Qvist, NORDUnet"
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Inatt kommer det att göras ett akut underhåll på Zoom-servrarna. Se nedan.
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tors 3 okt. 2024 kl 17:36
Subject: Scheduled Maintenance Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Scheduled Maintenance Reminder NORDUnet Status
*Title:* Emergency Zoom Maintenance
*Planned Start:* October 4, 2024 00:00 UTC
*Expected End:* October 4, 2024 04:00 UTC
*Affected Infrastructure*
*Components:* Zoom
*Locations:* On-Premise, Recording, Zone 2 - DK #1, Zone 3 - SE #1, Zone 6
- FI, Zone 7 - NO
*Details:*
We will perform emgergency maintenance on zoom servers. This may cause
disruption in ongoing meetings that need to re-established.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
Problemet är löst.
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tors 3 okt. 2024 kl 14:05
Subject: Status Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Incident Update NORDUnet Status
*Current Status:* Operational
*Started:* October 3, 2024 11:08 UTC
*Resolved:* October 3, 2024 12:05 UTC
*Affected Infrastructure*
*Components:* Zoom
*Locations:* Zoom Cloud, On-Premise, Recording, Zone 2 - DK #1, Zone 3 - SE
#1, Zone 6 - FI, Zone 7 - NO
*Update:*
The problem has been resolved.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
Det är för närvarande problem att ansluta till Zoom.
NORDUnet felsöker.
Mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tors 3 okt. 2024 kl 13:08
Subject: Status Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Incident Update NORDUnet Status
*Current Status:* Partial Service Disruption
*Started:* October 3, 2024 11:08 UTC
*Resolved:*
*Affected Infrastructure*
*Components:* Zoom
*Locations:* Zoom Cloud, On-Premise, Recording, Zone 2 - DK #1, Zone 3 - SE
#1, Zone 6 - FI, Zone 7 - NO
*Update:*
There are currently connection issues with Zoom. We're investigating.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
Hej alla!
Se nedan:
All,
The root cause for the problem has still not been identified.
Zoom has deployed a new version of the recorder with added debugging, to
help with the investigations.
The level of lost recordings is still in the 0,5% range.
Some of the lost recordings can be partly recovered manually, others are
just very late (+48Hours)
If there are *very* critical recording needs, it is recommended to use
local recording until the issue has been fully resolved.
Brgds,
Jørgen Qvist
MonZoom ika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Vi får fortfarande rapporter om förlorade cloudrecordings. Vår
rekommendation är att funktionen slås av på instans-nivå, dvs slå av
funktionen för alla för att minska risken att inspelningar går förlorade.
Zoom jobbar på att lösa problemet. Så fort vi har ny information så skickar
vi ut den.
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Fick precis den här infon från NORDUnet
Just a quick update.
There is no news since Sunday. There is a working session all day tomorrow,
that we hope will provide more information and help identify the cause.
Staus from yesterday is:
2024-09-16 1 meetings with recordings in processing out of 209 (0.48%)
Nästa uppdatering kommer torsdag.
För Sunets mediatjänster
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Vi hade hoppats ha information om Zoom Hybrid inför onsdagens mediamöte.
Dessvärre verkar det dröja så vi ses och pratar om hur NI använder Zoom hos
er med tanke på nya funktioner etc.
Har du något du vill dela med dig av, typ use case som andra kan ha nytta
av?
Vi ses på onsdag!
https://sunet.zoom.us/j/2772627210?pwd=MzR1eko3dTZ2MFF6ODFWdTFSQzg2dz09
mvh Monika, Peter och Markus
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Se info nedan:
Zoom on-premise cloud recording issue
During summer all of the NORDUnet Zoom on-premise platform was migrated
over to the new Zoom Node setup. Extensive testing was carried out as part
of this process.
There are things in the Node setup and backend that is having design flaws
regarding the scaling. We have together with Zoom identified 40+ issues or
features that needed to be addressed, but this has no impact on the
service, and we did not encounter any functional issues.
When utilisation increased after the summer holidays, we have encountered
an intermediate issue with the Zoom on-premise recording service. This has
resulted in the delay of the processing and making the recordings available
and has unfortunately led to the loss of recordings that cannot be
recovered.
During the period from August 27 to September 5, the average loss of
recordings was around 5%, in total around 250 recordings were lost.
The night between September 5 and 6 we implemented a workaround changing
the disc configuration on the recording server VM’s. After that the number
of lost recordings has dropped to less than 0,5% and a total of 5
recordings have been lost since.
The root cause for the problem has so far not been identified. There are
currently two different lines of investigations. We are in constant contact
with Zoom and provide them with all requested information.
In the new Zoom Nodes setup, the servers are running in locked down
containers that we do not have access to, making our ability to act
independently limited.
If there are critical recording needs, it is recommended to use local
recording until the issue has been fully resolved.
The Media support group has been kept informed on the Monday update
meeting, but we will start to inform more broadly on the situation.
We understand the frustration and concerns regarding the current situation
and NORDUnet apologies for and deeply regrets any inconvenience caused by
this.
För Sunets mediatjänster genom.
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
