Release Date: Tue, November 5, 2019 at 11:29 PM UTC
Release Author: Henrik Levkowetz

Trial bibxml3/ pages for drafts, and review ticket fixes

• Added a trial version of datatracker draft bibxml pages, located under
  /doc/bibxml3/; for example:
  https://datatracker.ietf.org/doc/bibxml3/draft-iab-fiftyyears-01.xml

• Added a management command to generate draft bibxml3 files.

• Fixed some type: ignore annotations

• Tweaked the submission tool to be more liberal with v3 xml submissions.

• Rewrote the ietf/bin/fix-ambiguous-timestamps command as a management
  command (fix_ambiguous_timestamps) and made it look at all instances of
  DateTimeField by means of model introspection.

• Merged in [16934] from sasha@dashcare.nl:
  Sort unassigned review requests by doc name by default.
  Fixes issue #2088.

• Merged in [16933] from sasha@dashcare.nl:
  Allow specifying review type for suggested reviews in LC and telechat. If
  a review is suggested on the manage unassigned reviews page, and the
  document is in both last call and telechat, the assign form now asks for
  the type of review that should be assigned. This commit also fixes two
  bugs in this process:

  • Comparisons in some cases between strings and integers
    (group/views.py:1485/1487)

  • Rejections when assigning suggested reviews, as they could be considered
    a newly opened request due to not having a pk (group/views.py:1508)

  Fixes issue #2119

• Merged in [16927] from sasha@dashcare.nl:
  Small cleanup from changeset [16924], see issue #2217.

Coverage

Release Date: Wed, October 30, 2019 at 1:34 PM UTC
Release Author: Henrik Levkowetz

Review tool tickets roundup, part 5; and other fixes

• Merged in [16926] from rjsparks@nostrum.com:
  Flipped a sense to put the grace period on the right side of today when
  sending review overdue reminders to team secretaries. Fixes #2816.

• Merged in [16937] from rjsparks@nostrum.com:
  Only show a blocked document once in the set of blocked documents on an
  ADs document page. Fixes #2818.

• Added mising GET parameters to the fragment cache in the agenda.ics
  template. Fixes issue #2817.

• Merged in [16924] from sasha@dashcare.nl:
  Allow submission of unsolicited reviews by secretaries.

  • For team secretaries, a button Submit unsolicited review will now
    appear next to Request review on the document's main page.

  • If the secretary is a secretary for multiple teams, they are taken
    through an intermediate page to select for which team they are
    submitting their review.

  • The form is similar (and using the same code) as the usual review
    completion, with a few extra fields for the review type and reviewer,
    which would usually already be known.

  • When submitting the review, a ReviewRequest and ReviewAssignment are
    automatically created. The assignment is then immediately closed in the
    usual way. Other workflows are unchanged.

  The issues with the review form in #2061 are slightly worse for the
  unsolicited review scenario, but that will be improved when #2061 is
  fixed. Fixes issue #2217.

• Merged in [16923] from sasha@dashcare.nl:
  Show doc authors and WG chairs in unassigned reviews views.
  Fixes issue #2087.

• Merged in [16922] from sasha@dashcare.nl:
  Added additional content validation for uploaded texts. Permitted MIME
  types are now text/plain, text/markdown and text/x-rst. This applies to
  all usages of get_cleaned_text_file_content(), including reviews, but also
  other similar places where text can either be written either into a
  textarea or uploaded. Fixes issue #2584.

• Merged in [16921] from sasha@dashcare.nl:
  Use a DBTemplate when a review assignment is mailed. The message sent to
  a reviewer when a new review is assigned to them, is now taken from a
  DBTemplate, allowing it to be customised per team. This includes a
  migration to create a default template, which looks the same as before.
  Fixes issue #2123.

Coverage

Release Date: Wed, October 23, 2019 at 3:57 PM UTC
Release Author: Henrik Levkowetz

Review tool tickets roundup, part 4; and other fixes

• Merged in [16904] from sasha@dashcare.nl:
  Always list all last call drafts in assignment summary. If a draft is
  scheduled for a telechat, but also assigned for LC review, it will now be
  listed both in the telechat section and under last calls. Fixes #2118.

• Merged in [16902] from sasha@dashcare.nl:
  Fixed a test failure in [16895]. This also makes a small text
  clarification, which was also the initial reason for the test failure.
  Related to #2526.

• Merged in [16895] from sasha@dashcare.nl:
  List previous reviews in assignments email.

  This includes a migration to update the database templates. Note that not
  all templates currently included them (secdir at least), and previous
  reviews are only listed in templates that included /* before. As
  before, previous reviews are only included if they are done by the same
  reviewer(s) as the current assignment.

  This also fixes a bug in database template 182
  (/group/defaults/email/open_assignments.txt) which referred to
  r.review_request..doc.rev and r.review_request..requested_rev in the
  template, and updates the test to use the current version of template 182.

  Fixes issue #2526.

• Merged in [16891] from sasha@dashcare.nl:
  Set default start date for UnavailabilityPeriod to today. Fixes #2198.

• Merged in [16883] from sasha@dashcare.nl:
  Do not allow reviewers to reject overdue reviews. If a review request is
  past the deadline, reviewers will no longer be able to reject the
  assignment. Fixes issue #2277.

• Merged in [16907] from rjsparks@nostrum.com:
  Correct construction for a reverse URL in an edge case. Fixes #2815.

• Merged in [16875] from rjsparks@nostrum.com:
  Adds docs with blocking positions to the ADs my-documents page. Fixes
  #2800.

• Merged in [16860] from rjsparks@nostrum.com:
  Be resilient to statistics queries that provide a state but not a
  substate. Fixes #2813.

• Fixed a couple of places where document alias.document was used instead of
  alias.docs.all().

• Merged in [16854] from rjsparks@nostrum.com:
  Remove some pre-ReviewAssignment refactor logic, and simplify what remains
  while still allowing a group secretary to not assign a row on the
  unassigned requests form after touching the control to assign a reviewer.
  Fixes #2812.

• Merged in [16853] from rjsparks@nostrum.com:
  Add submitted and posted dates to several ipr notification email messages.
  Fixes #2807.

• Merged in [16852] from sasha@dashcare.nl:
  Show stacked bar graph for in time and late reviews. On the stats page, a
  single stacked bar graph is now shown which combines in time and late
  reviews, replacing the separate graphs for these statistics. Tests for
  the charts are also expanded to validate the actual graph content for both
  stacked and non-stacked charts. Fixes issue #2331.

• Merged in [16850] from sasha@dashcare.nl:

  Close review assignments when reviewer is removed from team. If a
  reviewer is removed from a team, any accepted or assigned review
  assignments are set to rejected if the request deadline has not passed, or
  set to no-response when the request deadline has passed. Per [16819], if
  this scenario happens, and there are no other active or completed review
  assignments, the request is shown as an unassigned review. Fixes #2185.

• Merged in [16833], [16834], [16835], [16836] from rjsparks@nostrum.com:
  Changes to displaying the ballot_icon to account for IRSG ballots.
  Updates Grunt generated css and map files based on updated ietf.less to
  add styles for IRSG ballot positions. Updated copyrights in grunt
  generated bootstrap.js.

Coverage

Release Date: Mon, October 21, 2019 at 3:02 PM UTC
Release Author: Henrik Levkowetz

Use last-call@ietf.org for last-call announcements.

• Merged in [16824] and [16825] from rjsparks@nostrum.com:
  Introduced the use of last-call@ietf.org for last-call announcements,
  changing various email message templates and recipients to match.

• Merged in [16819] from sasha@dashcare.nl:
  Return review req to requested status if no review assignments are open.
  If a review assignment is rejected, withdrawn, marked no response, etc.,
  and this leaves a review request without any assigned/accepted/completed
  review assignments, return the request state to requested, which means
  it will be shown as an unassigned review in all interfaces.
  Fixes issue #2186.

• Merged in [16818] from rjsparks@nostrum.com:
  Make hybrid pdfs when converting ppt materials to pdf. Fixes #2769.

• Send output of the review reminders script to log, rather than stdout
  (since this is run from cron, stdout and stderr should be quiet if nothing
  unexpected happens)

• Added a guard against missing conflrev state in IESG agenda code.

Coverage

Release Date: Mon, October 21, 2019 at 1:58 PM UTC
Release Author: Henrik Levkowetz

• Merged in [16824] and [16825] from rjsparks@nostrum.com:
  Introduced the use of last-call@ietf.org for last-call announcements,
  changing various email message templates and recipients to match.

• Merged in [16819] from sasha@dashcare.nl:
  Return review req to requested status if no review assignments are open.
  If a review assignment is rejected, withdrawn, marked no response, etc.,
  and this leaves a review request without any assigned/accepted/completed
  review assignments, return the request state to requested, which means
  it will be shown as an unassigned review in all interfaces.
  Fixes issue #2186.

• Merged in [16818] from rjsparks@nostrum.com:
  Make hybrid pdfs when converting ppt materials to pdf. Fixes #2769.

• Send output of the review reminders script to log, rather than stdout
  (since this is run from cron, stdout and stderr should be quiet if nothing
  unexpected happens)

• Added a guard against missing conflrev state in IESG agenda code.

Coverage

Summary: Review tool tickets roundup, part 2
Release Date: Mon, October 7, 2019 at 3:06 PM UTC
Release Author: Henrik Levkowetz

• Merged in [16759] from rjsparks@nostrum.com:
  Restrict comments total on nomcom private index to comments on positions.
  Fixes #2792.

• Merged in [16705] from sasha@dashcare.nl:
  Send periodic reminders of open reviews every X days (opt-in).
  Fixes issue #2337

• Merged in [16704] from sasha@dashcare.nl:
  Send reminders before long UnavailabilityPeriods expire A notification is
  sent to the team secretary and reviewer, if an unavailability period
  lasting 30 days or more will end in 3 days. Fixes issue #2231.

• Merged in [16703] from sasha@dashcare.nl:
  Fix send-review-reminders and add it to daily cron. This fixes a syntax
  error and a Python 3 incompatibility, and adds send-review-reminders to
  the daily cron script. Related to issue #2231.

• Fixed an issue with the htmlization of links in release notes.

• Merged in [16607] from rcross@amsl.com:
  Fixed an issue where third session requests, which rerequire AD approval,
  get left with approved status, instead of scheduled, after Secretraiat
  sends out notifications of official schedule. Fixes #2765.

• Merged in [16701] from rcross@amsl.com:
  Added a migration to remove any break / registration sessions that have
  already been created for future meetings. Goes with changes for #2713.

• Added log.unreachable() for the code that creates unknown-email-*
  addresses. Related to issue #2696.

• Fixed a bug in gather_submission_submitter().

• Fixed an incorrect version number for mypy in requirements3.txt

• Removed some test assertions which are incorrect after the previous
  commit.

• From rcross@amsl.com: Don't copy breaks etc, when creating new Empty
  Schedule for meetings. Fixes issue #2713.

• Added a type hint stub file for ietf/api/init.py, to capture
  module variables created during module initialization.

Coverage

Summary: Review tool tickets roundup, part 1
Release Date: Tue, October 1, 2019 at 4:15 PM UTC
Release Author: Henrik Levkowetz

This release constitutes the first batch of fixes from the review tool
tickets work package. It also includes a few unrelated bugfixes.
From the commit log:

• Merged in [16688] from sasha@dashcare.nl:
  When completing a review, show which addresses it is posted to.
  Fixes issue #2478.

• Merged in [16680] from sasha@dashcare.nl:
  Made the review_completed mailtrigger configurable per team and review
  type. This includes a migration to change mailtrigger slugs to be up to
  64 characters instead of 32, as some slugs would not fit and require
  clunky abbreviations. A data migration creates triggers for existing
  teams, and they are also created on the fly if a trigger does not exist
  yet, providing a safe fallback for new review teams. The review_completed
  mailtrigger serves as the template for new triggers. This commit also
  includes tests for gather_address_lists(), as none existed.
  Fixes issue #2354.

• Merged in [16672] from sasha@dashcare.nl:
  Use mailtriggers to find destinations in review app As the review app has
  several conditionals that don't fit entirely well within mailtriggers, the
  templates use a bit of extra context to figure out who exactly to include.
  This also extends the tests for review, to check for correct recipients.
  It also adds a tiny feature to mailtrigger to entirely exclude certain
  addresses, as required by the review-generated mails. Fixes issue #2328.

• Merged in [16670] from sasha@dashcare.nl: Allow secretary to adjust date
  when completing a review. This also fixes other issues identified in
  #2590, around the modification of historical document events. The
  behaviour is now:

  • When the assigned reviewer posts a review, a single event is created,
    set to current date/time.

  • When the secretary records a review in the datatracker, they may set a
    different completion date, which is autofilled if an email is selected.
    One event is generated for the original completion date, and one for the
    secretary's action.

  • Each revision generates a new event, rather than updating previous
    existing events.

  Fixes issue #2590

• Improved the wrapping of long lines in emails sent by the datatracker:
  In wordwrap(), consider lines consisting entirely of some non-alphanumeric
  characters like ---- or === to be block (paragraph) separators. Fixes
  issue #2806.

• Merged in [16668] from sasha@dashcare.nl:
  Sort names alphabetically in review group history notes when changing the
  list of members. Fixes issue #2560

• Removed debug-related code from group edit clean_acronym().
  Fixes issue #2804.

• Merged in [16667] from sasha@dashcare.nl:
  Added an optional comment field for closing review requests.
  Fixes issue #2071

• Updated the docker files to build and run an image based on devuan:ascii
  instead of debian:jessie.

Coverage

Release Date: Mon, September 30, 2019 at 4:55 PM UTC
Release Author: Henrik Levkowetz

Test improvement: Starting to use Python type hints and mypy type checking

This release addresses two issues with the previous release, and as a result
of a test case failing to discover one of the issues, it also introduces the
use of Python type hints according to PEP 484. Type hints for common Python
library modules are provided by mypy, and type hints for Django are provided
by the django-stubs module. The problematic function that was used with the
wrong call signature has received a type annotation that will reveal similar
errors in the future.

This does not provide type annotation generally in the datatracker code;
that will require a lot of additional work (some of which can be automated),
but it gives us static type checking for Django and Python library calls,
and makes type hints useful when we provide them in datatracker code.

From the commit log:

• Fixed up mypy issues or added type:ignore comments as needed for a
  clean mypy run.

• Added a mypy .ini file

• Added mypy and django-stubs to Py3 requirements, and added a mypy test
  case.

• Changed the call signature of docevent_from_submission() to take an
  optional Person object, rather than person name. Added a type hint for the
  signature.

• Tweaked the condition for waiving submission email confirmation
  slightly.

Coverage

Summary: Draft submission tool changes
Release Date: Thu, September 26, 2019 at 6:42 PM UTC
Release Author: Henrik Levkowetz

This release contains two changes to the submission tool: During draft
submission, the email verification is skipped for logged-in users, and
author email addresses are now required for all authors, in order to avoid
author notification problems during IETF and RFC-Editor processing. Commit
log details:

• Changed draft submission to not require an email round-trip when the
  submitter is logged in. Added corresponding test cases.

• Added code to detect conflicting information for draft name, revision,
  and title when both XML and text versions of a document are submitted.

• Changed the draft submission form and views so that an email address is
  required for each author in order to complete self-service draft
  submission. It is possible to add these as metadata updates, but this will
  lead to submission through the secretariat (and the email addresses will
  still be required). Also updated related templates to make the handling of
  form section headings more consistent.

• Added some caching to /release/ pages. With now more than 350
  releases, the page response times had increased to typically 4s. Caching
  in view and template brought this down to ~160ms.

Coverage

Summary: Links to RFC text with in-lined errata. HTML for v3 XML draft submissions
Release Date: Fri, September 20, 2019 at 4:18 PM UTC
Release Author: Henrik Levkowetz

• Added conversion of version 3 XML draft submissions to v3 HTML, in
  addition to text. Added a link to HTML versions (when present) to the
  document status page, and renamed the link to the htmlized version to
  htmlized. Converted the test-file for XML submission tests to v3.
  Added test code to check that HTML is generated for v3 XML submissions.

• Added class bg-danger to submission error messages on the submission
  upload result page. The big Meta-Data errors found pane with red
  background is so noticeable that class text-danger (which gives bold
  text, but no change of background) is insufficient to stand out clearly.
  Adding bg-danger gives field error messages red background, too.

• Added a link from the datatracker RFC pages to the new RFC renderings
  with inline errata on the RFC-Editor web page (where available).

• Updated rfc-editor sync code to recognise the new Tooling Issue queue
  state.

• Updated ietf.sync.rfceditor.parse_index() to use the new rfc-index
  schema (changed placement of the page-count element).

• Added code to catch a form error regularly caused by form spam and
  return a failure instead of triggering a server 500.

Coverage