Summary: Removed Python 2.7 compatibility.
Release Date: Tue, April 14, 2020 at 7:45 PM UTC
Release Author: Henrik Levkowetz

This release removes python 2.7 compatibility code. It is a precursor to
conversion from Django 1.11 to Django 2.2 (via 2.0 and 2.1). The coming
Django 2.x compatible release will have a major version bump, to match.
Please note that there may however be some additional releases based on
Django 1.11 before we come to the first Django 2.x release.

• Merged in the py3clean work, which removes usage of six and future
  imports, along with other py2/3 compatibility code.

Coverage

Summary: Groundwork for upcoming automatic scheduling assistance
Release Date: Thu, April 9, 2020 at 4:25 PM UTC
Release Author: Henrik Levkowetz

• Merged in ^/branch/dash/automatic-scheduler@17395 from sasha@dashcare.nl,
  which adds groundwork for upcoming automatic scheduling assistance:

  ~ Added a management command to create a dummy IETF 999 meeting.

  ~ Added display of new constraints and joint sessions to agenda builder
  interface.

  ~ The new timerange, time_relation and wg_adjacent constraints, along with
  the joint_with_groups option, are now reflected in the special requests
  field. This allows them to be taken into account while scheduling
  sessions.

  ~ Clarified the wording in the session request form regarding conflicts
  with BOFs.

  ~ Added support for structured entry and storage of joint sessions in
  meetings:

  • Also adds additional tests for the SessionForm
  • Fixes a javascript error in session requests for non-WG groups,
    that could cause incorrect form behaviour.
  • Expands the tests added in [17289] a bit.

  ~ Added support for the timerange, wg_adjacent and time_relation
  constraints. This adds three new constraints to the database and
  relevant UIs:

  • timerange: "This WG can't meet during these timeframes"
  • wg_adjacent: "Schedule adjacent to another WG (directly following,
    no breaks, same room)"
  • time_relation: schedule the two sessions of one WG on subsequent
    days or with at least one day seperation

Coverage

Summary: Various meeting-related fixes and improvements
Release Date: Wed, April 8, 2020 at 4:51 PM UTC
Release Author: Henrik Levkowetz

• Merged in [17590] from rcross@amsl.com:
  Added support for variable length meetings to secr/meetings app.

• Changed the handling of some exceptions during draft submission to give
  user feedback rather than server 500 responses, in order to deal better
  with severely malformed drafts.

• Added a workaround for the current libmagic on OpenSUSE, which quite
  easily can mischaracterise text/plain documents as text/x-Algol68. Fixes
  issues #2941 and #2956.

• Added validation of the session duration in interim meeting requests, with
  added values in settings.py for min and max duration.

• Clarified the standalone XML draft submission requirements, and
  mentioned the xml2rfc switch to use for v3 sources.

• Changes to accept a wider range of URLs when displaying call-in links from
  the Session agenda_note and remote_instructions fields.

• Changed some fields to raw_id_fields in the MessageAdmin, for improved
  admin page load times.

• Added Remote instructions at the top of interim sesssion pages, and
  made the Meeting Details button available to the group chairs, not only
  secretariat.

Coverage

Release Date: Fri, April 3, 2020 at 10:04 AM UTC
Release Author: Henrik Levkowetz

Enhanced Upcoming Meetings page, and more

• Added links to agenda/materials pop-up, materials download, etherpad,
  jabber room, and webex call-in session for interims on the upcoming
  meetings page. With the earlier changes from [17555], this fixes issue
  #2937.

• Changed the IPR patent number regex to permit space between country
  code and serial number, and expanded on the help text for the IPR patent
  number field.

• Added verification of response data to IphoneAppJsonTests

• Merged in [17564] from pusateri@bangj.com:
  Added interim meetings to agenda.json API. Fixes #2946.

• Merged in [17562] from jennifer@painless-security.com:
  Add tooltips with doc name to updates and obsoletes links. Fixes
  #2866;

• Fixed a Py2/3 issue in the djangobwr's bower_install command

• Added a check for availability of bower in bin/mkrelease.

• Cleaned up the contributors list in bin/mkrelease a bit.

• Merged in [17557] from fenton@bluepopcorn.net:
  Provide more consistent links to people pages. Fixes #2918.

• Changed an obsolete document.href() to document.get_href(). Fixes
  issue #2945.

• Tweaked the upcoming calendar and calendar entries slightly, to render
  with times first and on two lines on narrow screens.

• Merged in [17555] from rjsparks@nostrum.com:
  Remove the not-quite-working customization widgets from
  /meeting/upcoming and /meeting/past. Simplify those views. Correct the list
  of sessions on those pages when one interim has more than one session.
  Fixes #2938. Partially addresses #2937.

• Prevent an exception on missing author.email.person when listing author
  emails.

Coverage

Summary: Fixes for meeting-related issues
Release Date: Fri, March 27, 2020 at 2:27 PM UTC
Release Author: Henrik Levkowetz

• Merged in [17543] from rjsparks@nostrum.com:
  Repaired construction of group_hierarchy used for the customisation
  controls at /meeting/upcoming. Fixes #2940.

• Merged in [17542] from rjsparks@nostrum.com:
  Show calendar entries on /meeting/upcoming in utc. Show end times.
  Partially addresses #2936.

• Don't show agenda buttons for Meetecho recordings (after the session
  concludes) if there isn't a Meetecho UrlResource. Fixes issue #2934

• Merged in [17538] from rjsparks@nostrum.com:
  Allow an out-of-area AD assigned as the AD for a WG to approve interim
  requests for that WG. Fixes #2930.

• Changed the object factory instances of nomcom private key and cert to
  be byte objects (matching the production settings), and fixed the issue
  with nomcom key handling under Py3 found by fenton@bluepopcorn.net. Did
  some renaming in nomcom/tests.py to better match setup/teardown function
  names to functionality.

• Merged in [17521] from housley@vigilsec.com:
  Improve performance of log.assertion() and log.unreachable()

• Merged in [17505] from housley@vigilsec.com:
  Improve performance of many document list pages

Coverage

Summary: IETF 107 code sprint
Release Date: Tue, March 24, 2020 at 5:53 PM UTC
Release Author: Henrik Levkowetz

This release contains datatracker bug fixes and enhancements from the
IETF-107 Code Sprint, our first virtual code sprint. It was a different
experience, and enjoyable despite not getting to sit down with a beer
together afterwards. A lot of good contributions were made; Thanks to
everyone who contributed!

• Merged in [17496] from rjsparks@nostrum.com:
  Remove the rest of the log.assertions checking that iesg_state existed in
  places we expected it to. Removed unnecessary imports.

• Changed the page for upcoming meetings to show the current IETF meeting
  for 7 days from its start date, while interims are shown for today and
  forward. Also changed the upcoming.ics calendar to show future sessions,
  even if the meeting to which they belong started in the past. This
  improves on [17518].

• Changed the starting point of display of upcoming meetings to be 7 days
  before today, rather than today, to let meetings linger a bit in the
  listing and iCalendar file after the meeting has started. Triggered by an
  observation from resnick@episteme.net about IETF 107 sessions disappearing
  from upcoming.ics on meeting week Monday.

• Merged in [17495] from rjsparks@nostrum.com:
  Removes a log.assertion() that was checking that we covered the edges when
  we changed documents to always have an iesg state.

• Merged in [17494] from rjsparks@nostrum.com:
  Use current email addresses when we have them when listing document
  authors. Fixes #1902.

• Merged in [17493] from mahoney@nostrum.com:
  Changed awkward IESG/IAB Nominating Committee names to just NomCom,
  updated a ref. Fixes #2860.

• Merged in [17492] from rcross@amsl.com:
  On session request form, made the Special Requests field smaller and
  display 200 character limit. Fixes #2875.

• Merged in [17491] from rcross@amsl.com:
  Prevent use of capital letters in group acronym. Fixes #2709.

• Merged in [17490] from rjsparks@nostrum.com:
  Basic regex validation on community rule entry form. Fixes #2928.

• Merged in [17489] from rcross@amsl.com:
  Removed redundant URL secr/groups/search because search page is
  available here secr/groups. Resolves issue with Add link. Fixes #2708.

• Merged in [17488] from rcross@amsl.com:
  Removed the drafts secretariat tool because this functionality is now
  provided by the core Datatracker. Moved ID reports to proceedings tool.
  Fixes #1655.

• Merged in [17487] from rjsparks@nostrum.com:
  Let chairs know what to do after material submission uploads have been
  cut off. Fixes #2887.

• Merged in [17486] from valery@smyslov.net:
  Added docker/run modifications to support Cygwin.

• Merged in [17484] from valery@smyslov.net:
  When requesting a new WG session, and retrieving information about the
  previous session, look back to the previous time the group met, instead of
  simply checking the previous IETF meeting and maybe not finding any
  information to retrieve.

• Merged in [17483] from peter@akayla.com:
  Changed things so that only WGs/RGs can be closed, per RJS. Fixes #1578.

• Merged in [17466] from rcross@amsl.com:
  Added a migration to cancel 107 sessions

• Added a check to see if any files matching the submitted draft name and
  revision already exists on disk in the active drafts or archived drafts
  directories, and if so reject the submission. Fixes issue #2908

• Made sure to strip possible mail header field values of whitespace
  before applying email.utils.unquite(). Resolution by kivinen@iki.fi,
  Fixes issue #2899.

• Merged in [17480] from rjsparks@nostrum.com:
  Show UTC times in interim announcements if the interim has a non-UTC
  timzone. Fixes #2922.

Coverage

Summary: Added agenda webex URL support, and meeting-related tweaks and bugfixes
Release Date: Fri, March 20, 2020 at 7:50 PM UTC
Release Author: Henrik Levkowetz

• Added webex URL to agenda.ics if Room.webex_url is non-empty. Fixes issue
  #2926.

• Added another check to the check_draft_event_revision_integrity management
  command, and refined it somewhat.

• Added a utility function to convert objects to dictionaries (for
  comparisons, for instance).

• Added a --dry-run option to bin/mkdevbranch, and added some exception
  handling.

• Help tablesorter see Month Year dates as dates with a hidden day digit.
  Fixes issue #2921.

• Refactored and extended check_draft_event_revision_integrity a bit.

• Tweaked bin/mkpatch some for -c handling

• Merged in [17442] from rjsparks@nostrum.com: Allow area groups to request
  interim meetings. Fixed #2919.

• Additional tweaks to bin/mkpatch; removing buggy -r option.

• Added automatic naming to bin/mkpatch when changeset or revision range is
  given.

• Added WebEx room resource name, query method and template logic to show
  WebEx room resources.

• Removed a debug statement

• Made links from agenda room names to floorplans conditional on the room
  having a floor plan set.

Coverage

Summary: Tweaks for wholly virtual meeting, for IETF-107
Release Date: Fri, March 13, 2020 at 2:48 PM UTC
Release Author: Henrik Levkowetz

• Added code to show a webex call-in button on the agenda page if the session
  agenda-note contains and IETF webex URL.

• Merged in [17425] from rjsparks@nostrum.com: Make required AD approval of
  virtual interims configurable. Fixes #2912.

• Added a management command to check draft event revision numbers. To be
  extended for other checks.

• Merged in [17419] from rjsparks@nostrum.com: Don't warn about idcutoff
  when the cutoff is after the meeting starts. Fixes #2907.

• Merged in [17418] from rjsparks@nostrum.com: Correctly represent cancelled
  sessions in ics files. Fixes #2905.

• Merged in [17396] from rjsparks@nostrum.com: Move charters for replaced
  groups to a new replaced state. Close any outstanding ballots on them.
  Fixes #2889, #2873, and #1286.

• Avoid trying to open meeting documents with empty .uploaded_filename.

• Added a progress bar for verbosity=1 of the community list index update
  command.

• Merged back fixes from production

• Corrected the extent of a try/except block, moving more code inside the
  block. Fixes a submission exception that should just be a document error
  reported back to the user.

• Added a guard against accessing attributes of None.

Coverage

Release Date: Sat, March 7, 2020 at 10:55 PM UTC
Release Author: Henrik Levkowetz

Submission API changes, Py2/3 transition fixes

• Added the ability to use the submission API with active secondary account
  email addresses. Fixes issue #2639.

• Tweaked the ReviewAssignmentAdmin, adding a raw_id_field.

• Replaced most cases of using of urlopen(), instead using the higher-level
  requests module where it simplifies the code.

• Added a data migration to fix up incorrect external URLs to mailarchive.

• Fixed a Py2/3 issue with review.mailarchive.construct_query_url().

• Renamed a migration to conform to migration naming conventions, using
  underscores instead of dashes in the name.

• Py2/3 compatibility tweaks for pyflakes.

• Changed some cases of urlopen() to use requests.get()

• Python3 is more ticklish about comparing strings to None than Py2. Fixed
  an issue with this in generate_sort_key() for document searches.

• Fixed a Py2/3 issue in the pyflakes management command, and tweaked the
  verbose output format.

• Merged back production changes to two scripts indirectly called by
  /a/www/www6s/scripts/run-ietf-report, through
  /a/www/www6s/scripts/run-report.

• Changed the release script to not pick up other email addresses than those
  of contributors from the release notes.

• Tweaked the check_referential_integrity management command verbose output.

Coverage

Release Date: Thu, February 27, 2020 at 7:56 PM UTC
Release Author: Henrik Levkowetz

Improved email handling, and roundup of Py2/3 conversion issues

This release provides a reworked email sending infrastructure so that all
emails sent by the datatracker, except for logging and failure emails sent
to ADMINS, are captured as a Message objects. Together with two new
management commands, this makes it possible to recover from email delivery
failures by re-sending any emails from the datatracker that would otherwise
have been lost.

This release also includes backporting Python 2/3 tranistion issues that
were initially patched in the production code during the recent server
transition, and a number of additional unrelated bugfixes.

From the commit log:

• Merged in [17336] from rcross@amsl.com: Add rfp-announce@ietf.org to
  Announcement app To: address list. Changed name of secr dashboard from WG
  Chair Dahsboard to IETF Dashboard.

• Merged in [17325] from rjsparks@nostrum.com: Allow review team secretaries
  and the secretariat to reset the next reviewer in queue for review teams
  using the RotateAlphabetically policy. Partially addresses #2879.

• Fixed an issue with some tests that mocked urlopen() responses, and
  tweaked error messages on bad results in sync/rfceditor.py

• Added charset decoding of data read from an urlopen() object in a few
  places, and removed some use-once functions that were now dead code.

• Moved code from a couple of simple urlopen() wrappers used only once into
  the calling scripts. Removed a case of Py2 virtualenv activation.

• Added management commands to show saved messages (unsent and otherwise)
  and to send or re-send saved messages as needed:

  • show_messages:

    Show outgoing messages that have been saved as Message objects. By
    default all messages from the last 2 weeks are shown. Selection can be
    made based on date and sent/unsent state. With the --pk option, only a
    list of primary keys are shown, otherwise, creation and send date,
    message-id, sender and primary recipients, and subject line is shown.

  • send_messages:

    Send (or re-send) messages saved as Message objects as outgoing emails.
    To show existing Message objects, use the show_messages management
    command. Messages to send can be indicateb by date ranges, a list of
    primary keys, or a list of Message-IDs. Unless the --resend switch is
    given, the inclusion of already sent messages in the date range or
    message lists will result in an error exit, in order to prevent
    inadvertent re-sending of message. Alternatively, the --unsent switch
    can be used to send only messages marked as not already sent from a date
    range or message list.

• Added a migration to set an explicit content-type for Message objects
  where missing.

• Added a migration to correct Message fields containing strings that were
  repr() of list instances instead of comma-separated email addresses.

• Added saving of outgoing emails as Message instances, with accompanying
  test.

• Did a slight refactoring of infer_message() and
  send_scheduled_message_from_send_queue(), and added setting of a
  Message.sent date when sent.

• Fixed an issue where the lookup of recognized country names during draft
  submission grabbed the unicode name instead of the ascii name for
  non-ascii country strings in XML submissions.

• Updated the list of utility python files that should be ignored by
  code coverage measurements.

• Fixed log.log() to hand on the right type (str) to underlying functions,
  irrespective of Python version.

• Fixed a test to be more specific when looking for a generated Message
  object, to make sure to get the right one.

• Added html unescape for email addresses from form data that might use html
  entities for angle brackets.

• Added a sent field to Message, to be able to track sent status for
  captured outgoing messages (which don't have a SendQueue instance).

• Fixed a number of bad Message.content_type settings when creating Message

• objects.

• Fixed a string vs list issue for a Message from form case.

• Added missing .as_strings() to some gather_address_lists() calls, in order
  to assign strings rather than lists to Message CharField instances. Fixed
  a Message content_type setting.

• More Py2/3 adaptations fetched from production.

• Converted send-scheduled-mail to py3

• Added a cron script to be called every 15 minutes, to have a place where
  send-scheduled-mail can live and be invoked with the right environment.

• Moved patch.py into our project tree -- it's not being maintained by the
  external maintainers any more.

• Merged in Python 2/3 changes pulled from the production server after the
  switch to python 3.

• Merged in cronjob script changes pulled from the production server after
  the switch to python 3.

• Added cleaning of the session request form's comments field, to convert
  any html entered to text. Related to [17322].

• Removed an instance of autoescape off for submitted meeting session
  comments in a template, since it is not needed in practice; in order to
  avoid an XSS injection opportunity.

• Updated the PLAN file, removing some completed actions.

• Changed some remaining cases of open() to io.open(), for Py2/Py3
  consistency.

• Tweaked settings.py to make django_extension commands available if
  installed.

Coverage