Summary: IETF 108 Code Sprint release 1, bugfixes and tweaks
Release Date: Tue, July 28, 2020 at 2:52 PM UTC
Release Author: Henrik Levkowetz

This release contains the first set of code sprint merges, and some
important fixes; additional code sprint contributions will come in
another release.

• Changed the URL to use for meeting attendees information in the new
  registration system.

• Merged in [18250] from jennifer@painless-security.com:
  Reject or require manual processing for submissions when inconsistent
  SubmissionDocEvent revs exist. Fixes #2909.

• Tweaked the messages from bin/rfc-editor-index-updates to be slightly
  more informative.

• Refined the code in ietf.sync.rfceditor that deals with errata to
  distinguish between rejected errata and other errata.

• Merged in [18270] from valery@smyslov.net:
  Added display of verified errata in red, not verified in orange in
  document lists. Fixes issue #2997.

• Merged in [18271] from valery@smyslov.net:
  Improved support of CygWin environment for the docker image.

• Added a few extra modules (in particular enscript and ghostscript,
  which is needed for the agenda pdf materials) to the docker/install-extras
  script.

• Added a workaround to display text/markdown as text in agenda popups.
  More work is needed to convert markdown to HTML for this view.
  See issue #3039.

• Improved the handling in the OpenID Connect userinfo response where there
  are multiple meeting registrations with different email addresses for the
  same person.

• Merged in [18285] from rjsparks@nostrum.com:
  Better handle materials with names that end with a string that looks
  like a version (-nn). May address #3040.

Coverage

Summary: Tweaks related to agenda display and docker-image build and start-up
Release Date: Sat, July 25, 2020 at 1:36 PM UTC
Release Author: Henrik Levkowetz

This is a small release to clear the slate before today's code sprint.

• Added a check for inallowable characters in slide titles at upload time.

• Added gather.town to the domains recognized as conference service
  domains.

• Swapped out the conference URL icon from fa-phone to fd-users, and
  changed the tooltip to Online conference.

• Added a setting for an explicit path to yanglint, in order to be able
  to detect its presence or absence.

• Reformulated a filter step in docker/updatedb to use a more commonly
  available command.

• Added yang/catalogmod to the yang module directories created by
  docker-init.sh.

• Fixed a problem with writing to /dev/stderr in some environments, in
  docker/updatedb.

• Added download progress information (where available) to docker/setupdb.

• Updated Dockerfile to use latest Devuan release (beowulf) as base
  image. Added syslog support.

• Tweaked the handling of yanglint settings.

• Made test parts that depends on yanglint conditional. Added some
  informational messages about tests skipped due to missing resources.

• Removed pip install into old virtualenv location from Dockerfile, and
  fixed an rsyslog config issue.

• Made the docker-container syslog startup less noisy. Refined the
  installation of virtualenv requirements to test for an existing django
  more correctly, and to use the repository requirements file if available.

• Updated font-datatracker, removing the fd-users and adding the fd-group
  glyph.

• Added individual ical links for meeting sessions on the agenda, and
  changed the phone handset icon to a group icon for the various online
  conference links.

• Merged in [18246] from rcross@amsl.com:
  Changed BOF or IRTF Sessions to just BOF Sessions at secretariat's
  request.

Coverage

Summary: Generalised external resourses for Groups, Documents, and People
Release Date: Wed, July 22, 2020 at 8:42 PM UTC
Release Author: Henrik Levkowetz

• Merged in the ExtResource work from rjsparks@nostrum.com, based on a
  feature request and contributions from rsalz@akamai.com. This provides
  external resource models for Person, Group, and Document models, as a
  generalisation of the previous related-URL concept. It has a consistent
  labelling and classification of URLs and other resources related to a
  Group, Document, or Person. The GroupURL (and similar) classes will be
  removed in a later step. Some DocumentURL instances (such as auth48 URLs)
  remain to convert to the new scheme, as they have come in with other
  merged-in work while the ExtResource work was in transit.

• Corrected and improved some language in the OIDC userinfo 'Request for
  Permission' view.

• Provided the new session attribute order_number to the template
  used to set the link to the Meetecho live video stream for sessions,
  in order to implement a requested change of URL format.

• Merged in [18198] from rjsparks@nostrum.com:
  Return the right reference for interim meeting materials.

• Merged in [18185] from rjsparks@nostrum.com:
  Only show sessions from the traditional week in the week view used
  for customized calendars. Partially addresses #3022 until a more general
  solution can be constructed.

• Merged in [18186] from rjsparks@nostrum.com:
  Let the weekview in the customized agenda show more than just the
  traditional meeting week. Fixes #3022.

• Merged in [18189] from jennifer@painless-security.com:
  Make docs in Auth48 state visually distinct. Fixes #2925.

• Fixed an issue for Session.order_in_meeting() if the session was
  cancelled.

• Merged in [18201] from rcross@amsl.com: Fixed an issue with the
  Secretariat Telechat app where the left pane would have missing sections.

• Merged in [18202] from rjsparks@nostrum.com:
  Serve older ietf meetings locally. Add tests to verify we have not set
  up a redirect to self.

• Merged in [18204] from rjsparks@nostrum.com:
  Recent past meetings are shown as past meetings, so sort them like past
  meetings. Fixes side-comment in #3101.

• Merged in [18205] from rjsparks@nostrum.com:
  Added rudimentary validation to the admin group form acronym field.
  Partially addresses #3026.

• Merged in [18206] from rjsparks@nostrum.com:
  Allowed modelforms, particularly the admin, to treat joint_with_groups as
  a optional field. Fixes #3029.

Coverage

Release Date: Wed, July 15, 2020 at 3:33 PM UTC
Release Author: Henrik Levkowetz

Extended support for the remote-instructions field to all session types;
Cookie settings and other changes for Meetecho integration; and bugfixes

• Updated the generate_name_fixture management command to include
  BusinessConstraint; necessary after the automatic meeting scheduler
  tests came in.

• Updated fixtures based on the updated generate_name_fixture command.

• Updated the Django patch for SameSite cookie settings to give a
  more informative error message from Django when the cookie setting
  is unacceptable.

• Merged in [18131] from rjsparks@nostrum.com:
  Add configuration and mechanisms to allow meeting materials to be
  served via a CDN. Fixes #2935.

• Merged in [18157] from jennifer@painless-security.com:
  Store Auth48 URL as DocumentURL and display in RFC-Editor state.
  Migrates old data when possible. Alternative to 17563. Fixes #2722.

• Fixed an UnboundLocalError issue.

• Merged in [18159] from jennifer@painless-security.com:
  Notify secretariat when conflict review/status change doc enters an
  announcement pending state. Fixes #2962.

• Tweaked the agenda HTML template, adding additional conditions
  (agenda-note, remote instructions) that will make session-related links
  show.

• Tweaked the list of days in secr session edit forms to allow setting
  the date to a day before the start of the meeting, necessary for IETF 108.

• Renamed some settings so as to start with the relevant app name
  (MEETING_*)

• Added support for a meeting session order_number (denoting first session
  for this group in the meeting, second session, etc.) when filling in URL
  templates with session info using the format filter. Used for the
  Meetecho video straming url for a session.

• Changed the logger level settings override in settings.py so as to not
  actually apply the example settings.

• Merged in [18162] from jennifer@painless-security.com:
  Notify IRSG when an IRSG ballot is created. Fixes #2978.

Coverage

Summary: Changes to cookie settings, yang checking improvement, and bugfixes
Release Date: Sat, July 11, 2020 at 8:46 PM UTC
Release Author: Henrik Levkowetz

• Added a warnings filter for a deprecated message triggered by a library
  module, to get rid of unhelpful deprecation messages.

• Increased the generate_schedule optimiser run count to 160, and tweaked
  some output messages, after several days of testing various run counts.

• Fixed a bad string formatting directive.

• Fixed an issue with secr.meetings.views.send_notifications() that could
  occur if a session didn't have a timeslot set.

• Merged in [18130] from rjsparks@nostrum.com:
  Fixed some test scaffolding that used non-existent keys (uncovered by
  recent improvements in sqlite3 integrity checking).

• Added a patch for Django to permit cookie samesite setting to be
  explicitly set to None.

• Added some cookie settings and changed others, in order to support
  login and operation from a Meetecho iframe.

• Added a guard against trying to use historic group parent information
  when parent is None

• Added support for pulling yang modules from yangcatalog.org, and use
  them when running yang checks on drafts.

• Changed the subclass of lxml.html.clean.Cleaner() to adapt to changes
  in the superclass in v4.5.2

• Added a new field Person.plain as a fallback for names where
  plain_name() produces the wrong result. Fixes issue #3020.

Coverage

Summary: Automatic meeting scheduler, improved support for remote instructions
Release Date: Wed, July 1, 2020 at 4:34 PM UTC
Release Author: Henrik Levkowetz

This release introduces the generate_schedule command, which will do auto-
layout of the schedule for an IETF meeting, with minimized conflicts. The
auto-scheduler output can also be used as input for further manual tweaks.
The goodness of the generated schedule very much depends on how complete the
input constraints are, and on the indicated cost of violating the various
constaints. It is expected that experience with using the output of the
auto-scheduler will lead to adjustment of the cost of violating various
constraints, and possibly also the introduction of new constraints.

There is also some other features, including support for the secretariat to
edit the remote_instructions field of sessions, which is necessary in order
to set remote call-in URLs for meeting sessions taking place in rooms that
don't already have a regular Meetecho channel.

From the commit log:

• Merged in the final part of the automatic scheduler from sasha@dashcare.nl.

• Renamed and adjusted a migration to avoid multiple migration leaf nodes.

• Fixed an issue with openid_userinfo() for persons with no email address.

• Renamed the automatic schedule generator command from schedule_generator
  to generate_schedule, so as to match our other generate_* management
  commands. Added some switches to the command, and added more information
  to the default output, providing information about number of optimisation
  runs and time consumed. Added a default value for the meeting option.

• Made it possible to provide a name to use for the generated schedule, and
  changed the autogenerated names to be easier to remember and type for
  later manual work based on an autogenerated schedule.

• Removed the command to purge request_profiler entries, given that
  request_profiler is incompatible with Django 2.2 and has been removed

• Fixed a bug in the caching of the agenda, where too much of the page was
  cached. Fixes issue #3008.

• Added a guard against trying to serialize session UTC start time for a
  meeting with unknown timezone.

• Renamed a test module to conform to our name convention ('test_*' to
  tests_*).

• Tweaked the HTML that puts session links and calendar links on the session
  details page so as to put them on the same line.

• Added some CSRF settings to make login from a meetecho iframe possible.

• Added remote_instructions as an editable field for misc. sessions in
  the secretariat app, in order to be able to set remote access URLs.

• Reworked the support for remote access urls in the agenda_note and
  remote_instructions fields of Session objects so as to accept any of a list
  of conference service domains as remote call-in URLs (i.e., not only webex).

Coverage

Summary: Django 2.2 upgrade
Release Date: Mon, June 29, 2020 at 10:56 AM UTC
Release Author: Henrik Levkowetz

• Changed deprecated load staticfiles to recommended load static

• Added a warnings filter. Removed the use request_profiler, which is
  not compatible with Django 2.2.

• Removed an unused parameter from submit.tests.do_submission_email().

• Updated requirements for Django 2.2.

• Fixed an issue where a session was saved without a type_id, found by
  the Django 2.2 checks. The code set the value just after the first save,
  and then did a second save, but this is 1) more costly, and 2) keeps an
  invalid session object in the database for a short time.

• Fixed a place where data provider for a POST in contained None, which
  cannot be serialized into POST data. Found by Django 2.2 checks.

• Django 2.2 does not wrap single queries in transactions, for
  performance reasons. This caused some template tags that did database
  lookups to trigger exceptions. Fixed by moving the lookups (which would
  not normally change between apache reloads) out from the template tag code
  to module scope. Adding new groups of type
  ['ag','area','team','dir','program'] will now require a reload to show up
  in the group menu.

Other changes:

• Changed skip messages from test suites to use print() instead of
  sys.stderr.write(), to match other output from the test runner.

• Updated MyPy to 0.770 and django-stubs to 1.5.0 for improved type
  checking of the Django interface calls.

Coverage

Summary: Django 2.1 upgrade, CodiMD support, bluesheet support for Meetecho, and bugfixes
Release Date: Sat, June 27, 2020 at 12:20 PM UTC
Release Author: Henrik Levkowetz

• Merged in [18036] from rjsparks@nostrum.com:
  Tweaked the title of the page used to complete reviews. Fixes #2767.

• Merged in [18037] from rjsparks@nostrum.com:
  Removed abandoned charters from any future telechats. Fixes #3003.

• Separated the session cache from the default cache, using slightly
  different settings, in order to not drop sessions when a new release is
  deployed.

• Added a default logger and level for django.security log items, to
  route them to our regular syslog. This will give us more information
  about failures to retrieve session info, and also things like 404
  responses, which will already be captured in the apache log but with
  less info. May need revisiting if it proves to cause excessive logging.

• Fixed a problem with the code to update a missing person record
  association in a meeting registration record.

• Fixed a problem with the handling of multiple registrations for the same
  person and meeting in api_new_meeting_registration().

• Added a switchover date (currently July 6th) for Etherpad to CodiMD use,
  and modified views and templates to provide links to codimd.ietf.org
  for meetings (including interims) after that date.

• Added an API endpoint for bluesheet information upload. Factored out
  code in common with manual upload_session_bluesheets(). Tweaked
  ietf.secr.proceedings.utils.handle_upload_file() to handle temp files in
  addition to HTTP file objects, to support the API endpoint.

• Updated bin/mkpatch to use a --name switch, and normalize the name to
  use - rather than _, and give more feedback.

• Added the meeting registration affiliation field to the OAuth2
  registration scope, so that Meetecho will have the information needed
  to build bluesheets.

• Merged in [18064] from rjsparks@nostrum.com:
  Changed the group edit field url regexp to accept the field names
  (specifically role names) we use by allowing hyphens.

Django 2.1 transition changes:

• Changed NullBoleanField() to BooleanField(null=True), according to the
  2.1 upgrade recommendations.

• Changed the removed django.contrib.auth.views.logout to LogoutView.

• Adjusted to the changed signature of admin.utils.get_deleted_objects()

• Adjusted the code to the attribute name change from ._size to .size in
  file upload objects.

• Updated various component requirements to match Django 2.1.

• The .render() method of widgets must now accept an additional keyword
  argument renderer; adjusted the code accordingly (including a library
  patch -- pull request also submitted).

Coverage

Summary: Test suite and OpenID improvements
Release Date: Mon, June 22, 2020 at 1:58 PM UTC
Release Author: Henrik Levkowetz

• In the OpenID registration scope handler, look for registration matches
  by email if no person link is available, in order to handle the case
  where somebody registered with a new email and only added it to the
  datatracker later.

• Added a test suite switch to optionally show logging output. Normally
  this is suppressed while running tests. In production, it goes to
  syslog's user facility. With --show-logging, it will go to the console
  when running tests.

• The registration system creates separate records for hackathon and
  session registration, but we want just one consolidated record.
  Changed the handling of notifications from the registration system to
  aggregate reg_type and ticket_type information in one record.

• With the draft submission cutoff date before a meeting set to the
  meeting start date (with the intention to have no blackout), the fact
  that submission re-open time is midnight local time led to a blackout
  period of a few hours after midnight UTC for meeting timezones with
  midnight later than UTC. Changed this to give no blackout time when
  cutoff-date==meeting.date.

• Merged in [18016] from rjsparks@nostrum.com:
  Ensure UTC time is shown throughout the interim request, approval, and
  announcement workflow. Fixes issue #2974.

• Merged in [18020] from rjsparks@nostrum.com:
  Construct Faker objects used for person name generation only once.
  Results in a roughly 20% speedup of the test-suite.

• Merged in [18021] from rjsparks@nostrum.com:
  Refactoring for a little less waste creating meeting/interim test data.

Coverage

Summary: Bugfixes
Release Date: Tue, June 16, 2020 at 9:16 PM UTC
Release Author: Henrik Levkowetz

• Fixed a bad regex in test-crawl

• Removed an unnecessary entry from the OIDC auth request arguments in
  test_oidc_code_auth().

• Reorganized the python logging setup slightly, and added a possibility
  to configure logging levels for individual loggers in settings_local.py

• Added a management command to list registered logging framewor entries
  (loggers), in order to see which can be configured in settings_local
  with the UTILS_LOGGER_LEVELS settings.

• Changed the OpenID profile data description to match what the
  datatracker actually shares. This eliminates mention of things the
  datatracker doesn't know, and doesn't share, such as gender and
  birth-date.

• The datatracker logs directly to syslog, but some python libraries send
  debug logging to the python logging module. Added a python logging syslog
  handler to the django logging setup, in order to make it easier to turn on
  built-in logging from external modules when needed.

• Improved the handling of logger enabling entries in UTILS_LOGGER_LEVELS
  to set a handler if none exists for a given logger.

• Changed the required version of pyflakes to one that supports ANNASSIGN
  (python annotated assignments, now used in the code).

• Added a --sprint switch to mkdevbranch to avoid switching on the
  absence of a developer name when creating sprint branches

• Merged in [17992] from rjsparks@nostrum.com:
  Improved the admin interface form validation for used_roles.

• Merged in [17993] from rjsparks@nostrum.com:
  Made test_interim_send_announcement a little more time_zone aware.

Coverage