Hej alla!
Se nedan.
Mvh
Monika
Skickat från min iPhone
Vidarebefordrat
> Ämne: [Mediagroup] Re: Zoom on-premise recording issue update 20241101
>
> All,
>
> Unfortunately the change to the storage hardware and logical infrastructure did not resolve the problem.
>
> We did however uncover a problem where the recorders are started before the NFS mounts are fully setup. We have tried to implement a fix for this and we are currently seeing an improvement of the situation.
>
> We will monitor this closely.
>
> Next update Friday November 15, late as I will be travelling back from the US.
>
> Brgds,
>
> Jørgen Qvist
>
>
> Please read NORDUnets Data Protection Policy - https://nordu.net/privacy-policy/ & Legal Notice - https://nordu.net/legal-notice/
>
>
> _______________________________________________
> Mediagroup mailing list -- mediagroup(a)lists.nordu.net
> To unsubscribe send an email to mediagroup-leave(a)lists.nordu.net
Hej!
Det här kom i fredags:
All,
Unfortunately the new version of the Recording Connector did not resolve
the problem.
Zoom is continuing to work on a solution at their end.
NORDUnet has in a maintenance window Thursday October 31, made changes to
the Recording Nodes storage hardware and logical infrastructure, for two of
the operational Zoom Zones. We will monitor the impact before making
further changes.
Next update Friday November 8.
Brgds,
Jørgen Qvist
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
All,
enclosed is the Zoom Root Cause Analysis (RCA) of the issue encounter
between 17 and 21 UTC on 20241022.
The effect of the issue was that users on a subset of sub-accounts could
not start/join meetings and instead received 3010 errors.
The issue was triggered when NORDUnet was doing planned maintenance on a
Zoom Zone, removing that Zone from all sub-accounts.
Zoom had recently deployed a new security feature that had a flaw which
then returned a completely empty Zone list to these sub-accounts, that then
had no active Zoom Zone to connect to.
The workaround was to disable the security feature which restored
operations.
A fix has been deployed that permanently has resolved the problem.
Full Zoom RCA enclosed.
NORDUnet regrets the problems encountered due to this.
Brgds,
Jørgen Qvist
Please read NORDUnets Data Protection Policy -
https://nordu.net/privacy-policy/ & Legal Notice -
https://nordu.net/legal-notice/
_______________________________________________
Mediagroup mailing list -- mediagroup(a)lists.nordu.net
To unsubscribe send an email to mediagroup-leave(a)lists.nordu.net
Hej alla!
Se mejl från NORDUnet nedan. Om det efter helgen dyker upp problem så låt
oss veta det via media(a)sunet.se
All,
Finally good news.
A new version of the Recording Connector has been made available and will
be deployed during the weekend. The new version includes the fix for the
recording problem that we have been struggling with.
We are however reluctant to say that the problem has been fully fixed by
Monday.
During the next weeks we will monitor closely to verify that no recordings
are being lost and that the recording flow is stable.
If you encounter any issues please let us know immediately.
Thx,
Brgds,
Jørgen Qvist
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej!
Vi har för närvarande problem med Zoom och felsökning pågår.
mvh
Monika
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tis 22 okt. 2024 kl 19:17
Subject: Status Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Incident Update NORDUnet Status
*Current Status:* Service Disruption
*Started:* October 22, 2024 17:17 UTC
*Resolved:*
*Affected Infrastructure*
*Components:* Zoom
*Locations:* On-Premise, Zone 3 - SE #1
*Update:*
There are currently connection issues with Zoom. We're investigating.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
All,
some good news. Zoom believe that they have identified the root cause of
the problem.
It has to do with that the recording connector needs to migrate files
during processing.
(If the disk is busy at the time the file movement might fail and this is
resulting in a “dirty” file, and this also affects subsequent files and is
the reasoning for the generic slowness.)
Zoom is working on a new release of the recording connector, that will be
deployed as soon as available.
Brgds,
Jørgen Qvist
Det kan ta upp till 48 timmar innan en recording dyker upp men *vår
rekommendation (och NORDUnets) är att stänga av funktionen tills problemet
är löst. *Vi har kommunicerat detta i omgångar, men det trillar fortfarande
in ärenden om förlorade inspelningar.
Vill ni vara helt säkra på att det fungerar så stäng av på kontonivå. Vi
vet att både NORDUnet och Zoom jobbar på problemet.
Alltså, stäng av funktionen för cloudrecording i Zoom tills problemet är
löst.
Mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej!
Detta kom nu på morgonen.
"All,
There is nothing new in regards to the status, the root cause has still not
been identified. There is a lot of effort going in to finding the cause of
the intermediate problem, both by Zoom and NORDUnet and we have found other
things that have been fixed along the way.
There was an issue with Zoom that affected the Zoom users in Finland and
parts of Sweden Thursday afternoon. This was cased by some local VM disks
running full due to the increased logging enabled to debug the recording
issue. The problem was quickly resolved.
Brgds,
Jørgen Qvist, NORDUnet"
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Inatt kommer det att göras ett akut underhåll på Zoom-servrarna. Se nedan.
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tors 3 okt. 2024 kl 17:36
Subject: Scheduled Maintenance Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Scheduled Maintenance Reminder NORDUnet Status
*Title:* Emergency Zoom Maintenance
*Planned Start:* October 4, 2024 00:00 UTC
*Expected End:* October 4, 2024 04:00 UTC
*Affected Infrastructure*
*Components:* Zoom
*Locations:* On-Premise, Recording, Zone 2 - DK #1, Zone 3 - SE #1, Zone 6
- FI, Zone 7 - NO
*Details:*
We will perform emgergency maintenance on zoom servers. This may cause
disruption in ongoing meetings that need to re-established.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
Problemet är löst.
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tors 3 okt. 2024 kl 14:05
Subject: Status Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Incident Update NORDUnet Status
*Current Status:* Operational
*Started:* October 3, 2024 11:08 UTC
*Resolved:* October 3, 2024 12:05 UTC
*Affected Infrastructure*
*Components:* Zoom
*Locations:* Zoom Cloud, On-Premise, Recording, Zone 2 - DK #1, Zone 3 - SE
#1, Zone 6 - FI, Zone 7 - NO
*Update:*
The problem has been resolved.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
